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Abstract 

The categorical compositional distributional 
model of Coecke et al. (2010) provides a lin¬ 
guistically motivated procedure for computing 
the meaning of a sentence as a function of the 
distributional meaning of the words therein. 

The theoretical framework allows for reason¬ 
ing about compositional aspects of language 
and offers structural ways of studying the un¬ 
derlying relationships. While the model so 
far has been applied on the level of syntac¬ 
tic structures, a sentence can bring extra in¬ 
formation conveyed in utterances via intona- 
tional means. In the current paper we ex¬ 
tend the framework in order to accommodate 
this additional information, using Frobenius 
algebraic structures canonically induced over 
the basis of finite-dimensional vector spaces. 

We detail the theory, provide truth-theoretic 
and distributional semantics for meanings of 
intonationally-marked utterances, and present 
justifications and extensive examples. 

1 Introduction 

Distributional models of meaning, in whieh a word 
is represented as a high dimensional veetor of con¬ 
textual statistics in a metric space, provide a con¬ 
vincing framework for lexical semantics that has 
been found useful in a number of natural language 
processing tasks (Schiitze, 1998; Landauer and Du- 
mais, 1997; Manning et al., 2008). Despite their 
success at the word level, the underlying hypothe¬ 
sis of these approaches does not naturally scale up 
to phrases or sentences due to the infinite capacity 
of language to produce new meanings from a finite 
vocabulary and a set of grammar rules. 


Coecke et al. (2010) provide a solution to the 
problem by noticing that the category of finite¬ 
dimensional vector spaces and linear maps is ho¬ 
momorphic to a grammar expressed as a pregroup 
(Lambek, 2008); specifically, bofh share compacf 
closed sfrucfure (Kelly, 1972). In pracfice fhis 
means fhaf any grammafical derivation based on fhe 
fype-logical identifies of fhe individual words in a 
senfence can be franslafed fo a (mulfi-)linear map 
which, when applied on fhe vecforial represenfafions 
of fhe words fherein, resulfs in a sentence vector. 
The grammafical fype of a word determines fhe vec- 
for space in which fhis word lives. Taking nouns to 
be simple vecfors in a basic vector space N, an ad¬ 
jective, for example, becomes a linear map N ^ N, 
or equivalenfly, a mafrix in N ^ N', furfhermore, a 
fransifive verb is a bi-linear map A (g) A —)• S, living 
in iV (g) 5 (g iV. Composition lakes fhe form of ten¬ 
sor conlraclion, which is a generalizalion of mafrix 
mulfiplicalion fo higher order lensors. 

In general, fhe model resembles a quanlilafive 
linear-algebraic version of fhe formal semantics ap¬ 
proach (Monlague, 1970), in fhe sense lhal synlax 
striclly guides fhe semantic composifion. Inferesl- 
ingly, synlax seems fo co-exisl wilh a dislincf sfruc- 
lural layer, fhe purpose of which is to optimize fhe 
message lhal an ulferance conveys. This aspecl is 
known as information structure, and al fhe phrase 
or senfence level is expressed as a disfincfion be- 
Iween a Iheme pari (informalion lhal is generally 
agreed fo be known to bofh of fhe inlerlocufors) and 
a rheme part—information lhal is new for fhe ad¬ 
dressee. The exacl relation lhal holds befween syn- 
lacfical and information struclure is an inleresling 
and conlroversial lopic. For example, a Iheme does 
nol have to comprise a valid grammafical consliluenf 


in the strict sense of the term, as it is evident in the 
following example: 

(1) Q: Do you need anything? 

A: [I would likejj [some tea]R 

The distinction between a theme and rheme is de¬ 
noted by the presence of a boundary that can be 
expressed by phonological, morphological or even 
syntactical means, depending on the language. Fur¬ 
thermore, the presence of such boundaries suggest 
the existence of a distinct composition operator re¬ 
lated to information structure and different than the 
one that would be normally used for syntax. 

In this paper we extend the categorical model of 
Coecke et al. (2010) in a way to accommodate an 
information structure layer of composition. In or¬ 
der to achieve this, we model intonational bound¬ 
aries (the devices for defining information struc¬ 
ture in English) by using the multiplication part of 
the Frobenius algebra that is canonically induced 
over any vector space with fixed basis, in order fo 
endow equal confribufion of fhe fheme and rheme 
on fhe vectorial represenfafion of a senfence, fhus 
pulling emphasis on fhe appropriate pari. The re- 
sulling model can be seen as conlaining Iwo types of 
composition operators: fhe usual tensor conlraclion 
for accommodating synlax, and fhe Frobenius mulli- 
plicalion for accommodating informalion slruclure. 
We discuss fhe implicalions in terms of fhe resulling 
vectorial represenlalions for phrases and senlences, 
and provide connections wilh exisling models from 
fhe currenl liferalure of composifional dislribulional 
semanlics. Various examples demonsfrafe fhe poten¬ 
tial of fhe model. 

2 Categorical compositional distributional 
semantics 

The calegorical model of Coecke el al. (2010) as¬ 
signs semantic represenlalions to phrases and sen¬ 
tences of language, based on Iheir grammatical 
slruclure and Ihe semantics of individual words. In 
ils mosl abslracl form, Ihis model can be expressed 
in terms of a slruclure-preserving passage belween 
grammar and meaning: 

T : Grammar —)■ Meaning 

Given a sequence of words wi • • • w„, ils categor¬ 
ical meaning is defined to be: 

|wi ■ • • w„]| := J-(a)(|wi], • • • , |w„]|) (1) 


Here, a is derived from Ihe grammatical relation¬ 
ships amongsl Ihe words in Ihe sequence. This no¬ 
tion can be formalised in a coherenl way, if bolh Ihe 
grammar and Ihe meaning are expressed in a high 
level logical slruclure, referred to by compact clo¬ 
sure. Fambek’s pregroup algebras (Fambek, 2008) 
and vector space dislribulional semantics are exam¬ 
ples of compacl closed slruclures. Stipulating lhal 
Ihe grammar is expressed in a pregroup algebra and 
lhal Ihe meaning of words are vectors conslrucled 
using Ihe dislribulional hypolhesis (Harris, 1968), 
Eq. 1 gels a more concrete form: ^ 

Wi • ■ • := J'(q;)(\^i (g) • • • (g) w^„) (2) 

In Ihe proceeding subsections we make Ihese no¬ 
tions precise and provide inluilions and examples. 

2.1 Pregroup grammars 

A pregroup grammar is a pregroup algebra, linked to 
Ihe vocabulary of a language via Ihe notion of a type 
dictionary. We define Ihese slruclures below. 

A pregroup algebra is a partially ordered monoid 
where each elemenl has a left and a righl adjoint II 
is denoted by a luple (P, <, •, 1 , (—)^ {—Y), where 
(P, <) is a partially ordered sel, and • is a monoid 
multiplication wilh 1 as ils unit For each elemenl 
p G P Ihere are G P, referred to by p’s left 

and right adjoinls, satisfying Ihe following four in¬ 
equalities: 

p • p’’ < 1 < p'" • p p^ • p < 1 < p • p^ 

When a pregroup algebra is generated over a base 
sel B, il is denoted by P{B). Given Ihe vocabu¬ 
lary of a language S and a sel of ils basic gram¬ 
matical types B, a pregroup grammar is a relation 
D QT, X P{B) lhal assigns grammatical types from 
Ihe pregroup algebra P{B) to Ihe words of Ihe vo¬ 
cabulary S. Such a pregroup grammar is denoted by 
P(P,S). 

As an example, suppose B = {n, s}, where n 
slands for a well-formed noun phrase and s for a 
well-formed sentence. Suppose furlher lhal S = 
{Mary, snores, likes, musicals}. The pregroup dic¬ 
tionary consisls of Ihe following sel: 

{(Mary, n), (snores, rf s), (likes, iPsiY), (musicals, n)} 

*One can translate the types from other type logics, such as 
the syntactic calculus and CCG to pregroups and carry on with 
the same calculations. There is also recent work that directly 
assigns vector semantics to CCG (Lewis and Steedman, 2013). 



One says that a sequence of words W 1 W 2 • • • 
for Wj G S forms a grammatical sentence, according 
to a pregroup grammar P{B, S), whenever we have: 

ti ■ t2 ■ ■■ ■ ■ tn < S 

for (wjjtj) G D. The above inequality is often re¬ 
ferred to by grammatical reduction. For example, 
‘Mary likes musicals’ is a grammatical sentence, 
since we have the following reduction: 

n • n’’ • s • • n < 1 • s • 1 = s (3) 

2.2 Distributional models 

The only piece of information provided by a deriva¬ 
tion like the one in Eq. 3 is whether the sentence 
in question is well-formed or not. Furthermore, we 
are unable to distinguish between words of the same 
type. Distributional models of meaning provide a 
solution to these problems by following the distri¬ 
butional hypothesis (Harris, 1968), which states that 
semantically similar words must appear in similar 
contexts. Hence, the semantic representation of a 
word can be given in terms of its distributional be¬ 
haviour in a large corpus of text. In its simplest form, 
a word vector is comprised by numbers that show 
how many times the target word co-occurs with ev¬ 
ery other word in a selected subset of the vocabulary 
(usually the most frequent content-bearing words). 
This allows the representation of words as points in 
some high dimensional space, where semantic relat¬ 
edness can be measured (usually by cosine distance) 
and evaluated. For a concise introduction to dis¬ 
tributional models, see (Turney and Pantel, 2010). 
We will now proceed to show how the quantitative 
approach of distributional models can be combined 
with the compositional model of Section 2.1 into a 
unified account. 

2.3 Categorical generalization 

The theory of categories generalises algebraic con¬ 
structions to categorical ones (Mac Fane, 1971). 
Herein, instead of sets, functions or relations, one 
has objects A,B and morphisms f: A ^ B. The 
generalised binary operation over these is referred 
to by a product. Posing different conditions on the 
objects, morphisms, or the product results in differ¬ 
ent kinds of categories. A monoidal category has a 
product with a unit I, that is A® / = I ® A = A. 
These categories are generalisations of partially or¬ 
dered monoids: elements of the partial order be¬ 


come objects of the category and the partial order¬ 
ings between them become morphisms. Further¬ 
more, compact closed categories are generalisations 
of pregroups, where the adjunction inequalities cor¬ 
respond to the following e and tj morphisms: 

e^: A 0 A'' ^ I if : I ^ A^ ® A 

f : A'- ® A-1- I if : I ^ A® A^ 

These maps needs to adhere to four axioms, re¬ 
ferred to as yanking equations, which ensure that all 
relevant diagrams commute. 

The importance of the theory of categories for this 
paper is that finite-dimensional vector spaces and 
linear maps also form a compact closed category, de¬ 
noted by FVect. Herein, objects are vector spaces, 
morphisms are linear maps, and the product is the 
tensor product between vector spaces whose unit is 
the scalar field of fhe vector spaces, in our case, real 
numbers (M). In fhe presence of a fixed basis (which 
is fhe case we are inferesfed in) fhe adjoinfs become 
idenfify, fhaf is we have V'' = V^ = V, for a vector 
space V spanned by {1^*}*. As a resulf fhe four e 
and p maps reduce fo fwo: 

e:H(8)H— p: R ^ V 0V 

The e map fakes fhe inner producf of fwo vectors 
and fhe p map produces a diagonal mafrix. The facl 
fhaf bofh pregroup algebras and vector spaces form 
compacf closed cafegories allows us fo develop a 
sfrucfure preserving passage befween fhe fwo mafh- 
emafical sfrucfures, fhus enabling us fo bridge fhe 
grammatical sfrucfure fo disfribufional semantics. 

2.4 From grammar to distributions 

A structure preserving passage from grammatical 
structures (in the form of a pregroup grammar) to 
semantics (in the form of vector spaces) is given by 
a map denoted as follows: 


P : F(E,B) ^ FVect (4) 

This is a strongly monoidal passage, which means 
that it has the following compositional properties for 
juxtapositions of types in a pregroup grammar: 

P(l) = M (5) 

B(p-q) = P(p)0P(q) (6) 

P(p’')=P(f) = P(p) (7) 

P(p<g) = P(p)^P(q) (8) 



On the level of basic types we assign a vector 
space to each basic type, that is, T{n) = N and 
-F(s) = S. As a result of the above assignments, 
words that have simple types, for example noun 
phrases, will become vectors in vector space N. 
Words that are functions of one argument become 
matrices, e.g. intransitive verbs with type rf ■ s 
are elements of iV (g) 5; and words that are func¬ 
tions of two arguments, e.g. transitive verbs with 
type rf ■ s ■ v}, become tensors of order 3, living 
in A" (g 5 (g for the specific case. The grammati¬ 
cal reductions are translated to compositions of mor- 
phisms, and in particular e-maps. 


Mary snores = • n’’• s) (Mary (g snores) 

= (e)v (g Is)(Mary (g snores) 

A simple computation shows that the above is 
equal to Mary x snores; similarly, for the meaning 
of a transitive sentence we obtain: 


Mary likes musicals = Mary x likes x musicals 

Note that tensor contraction (in spaces with fixed 
basis) is associafive, so fhere is no need fo keep frack 
of brackefs in fhe above. The sifuafion is similar fo 
pregroups, where fhe monoid mulfiplicafion is again 
associative. 

2.5 Frobenius algebras 

Compacf closed categories on fheir own do nol have 
much sfrucfure: fhere is a binary operation and fhe 
maps e and tj. The expressive power of fhese caf- 
egories can be increased using Frobenius algebras. 
We define fhese below. 

Given a compacf closed cafegory C, an objecf 
A € C has a Frobenius sfrucfure on if if fhere ex¬ 
ist fhe following morphisms: 

A:A^Ag)A l: X ^ I 

H:X(^X^X C:I^X 

These have fo satisfy cerfain conditions, fhe mosf 
imporfanf fo us being fhe Frobenius condition: 

(t* ® lx) o (lx ® A) = Ao^ = (lx (g/i) o (A 0 lx) 

Vecfor spaces wifh fixed basis do have such sfruc- 
fures over fhem, generally referred fo by copying and 
merging. For G F, u; G F g) F, we have fhaf 


A(lt) Gf^gFisa diagonal mafrix whose diag¬ 
onal elemenfs are weighfs of it, and p,(w) G F is 
a vecfor consisting only of fhe diagonal elemenfs of 
w. 

These sfrucfures have been used in previous work 
fo encode lower dimensional verb mafrices info 
higher dimensional tensors (Karfsaklis el ah, 2012; 
Karfsaklis el ah, 2014) and fo pass fhe informalion 
around sentences wifh relative clauses by copying 
and merging (Sadrzadeh el ah, 2013; Sadrzadeh ef 
ah, 2014). 


2.6 Graphical calculus 

In fhe presence of higher order tensor producf 
spaces, calculalions can become quite complex. The 
formalism of compacf closed categories and Frobe¬ 
nius sfrucfures is complele wifh regard fo a graphical 
calculus (Selinger, 2011) lhal simplifies fhe compu- 
fafions fo a greal exlend. We briefly overview fhe 
main componenfs of Ibis language. 

Objecls are depicled by lines and morphisms by 
boxes. Tensor producls belween objecls and mor¬ 
phisms are given by juxlaposilion of fheir diagrams, 
while composition of morphisms amounls fo con- 
necling oulpuls fo inpuls. Examples are as follows: 


A 



The e maps are depicled by cups, p maps by caps, 
and yanking by fheir composition and slraighlening 
of fhe slrings. For inslance: 


A‘ A 




A A‘ 


I 

X A 


X = 


A 


The diagrams corresponding fo fhe Frobenius 
morphisms are as follows: 

(M,C)='Sj>-^J (A,t)=^^| 

wifh fhe Frobenius condilion being depicted as: 


The defining axioms guarantee lhal any piclure of 
a Frobenius compulation can be reduced fo a normal 
form (so-called a “spider”) lhal only depends on fhe 
number of inpul and oulpul slrings of fhe nodes: 




Elements within the objeets (for the ease of veetor 
spaees, veetors) are depieted by morphisms from the 
unit. These are shown by triangles with a number of 
strings emanating from them. The number of strings 
denotes the order of the tensor; for instanee, the dia¬ 
grams for if £ V,v' £ V®W, and v" £ V®W®Z 
are as follows: 

V V w vwz 

3 Information structure and intonation 

The term information structure eolleetively refers to 
teehniques that aim to enhanee the eommunieation 
between two interloeutors in order to optimize the 
eonveyed message for the benefit of the addressee 
(Chafe, 1976). One sueh teehnique, for example, is 
to emphasize a partieular part of the utteranee that 
is important for the listener by ehanging the spoken 
piteh: 

(2) Q: What does Mary like? 

A: Mary likes MUSICALS 

The emphasis imposes a speeifie information 
strueture to the uttered sentenee, essentially splitting 
it in two parts: The part in upper-ease above is what 
Steedman (2000) ealls rheme —the information that 
the speaker wishes to make eommon ground for the 
listener; the rest of the sentenee, i.e. what the lis¬ 
tener already knows, is ealled theme. The question 
in (2) puts the listener in a speeifie affenfional sfafe, 
in fhe eonfexf of whieh an answer sueh as: 

(3) A: #Mary likes musieals 

will be infelicitous, fhaf is, nol eompafible wifh fhaf 
sfafe. 

The disfinefion befween fheme (or fopie) and 
rheme (or eommenf) has greaf signifieanee from an 
information sfruefure poinf view, sinee if defines a 
generie shape for fhe senfenee fhaf direefly refleefs 
fhe affenfional needs of fhe addressee. A furlher di¬ 
mension fhaf ean be found in bofh rheme and fheme 
disfinguishes befween the focus, fhaf is, fhe speeifie 


word fhaf reeeives mosf of fhe infonafional empha¬ 
sis, and fhe baekground, whieh eonsisfs of fhe resf of 
fhe words in fhe speeifie fexf segmenf. Nofe fhaf in 
eonfrasf fo rheme/fheme disfinefion, foeus and baek¬ 
ground seem fo operafe af fhe lexieal level.^ 

Furfhermore, we should poinf ouf fhaf alfhough 
fhe examples we use in Ibis paper are mainly based 
on quesfion/answer dialogues, fhis is nol by any 
means fhe only ease where fhe presenee of a speeifie 
information sfruefure ean be useful. For example, 
eonsider fhe dialogue: 

(4) -I fhink Mary likes jazz. 

-Mary likes MUSICALS. 

Information sfruefure ean be expressed in differ- 
enl ways fhaf may vary from language fo language. 
In English, for example, fhe means for defining in¬ 
formation sfruefure is intonation: varialions of spo¬ 
ken pileh, fhe purpose of whieh is fo emphasize parls 
of fhe ulleranee fhaf mighf be imporlanf for fhe eon¬ 
veyed message, as we saw above in our examples. 
However, in ofher languages sueh as Japanese or 
Cantonese, fhe infonafional boundaries ean be also 
speeifieally marked by morphologieal deviees, e.g. 
speeial parfieles (Fery and Krifka, 2008). Finally, 
fhe posilion of a lexl segmenf in a senfenee ean also 
be an indieafion of ifs informalion-slruelural role. In 
English, for example, fhemes lend to appear af fhe 
beginning of a elause. 

In Ibis paper we eoneenlrafe on fhe senlenee-level 
disfinefion befween rheme and fheme. 

4 Grammar and intonation 

The presenee of a dislinel layer of informalion slrue- 
lure fhaf seems fo eo-exisf wifh fhe grammalieal 
sfruefure of a senfenee, poses fhe inleresling ques¬ 
tion regarding fhe exael relalionship fhaf holds be- 
Iween Ihose differenl slruelural aspeels. For exam¬ 
ple, allhough fhe lexl segmenf “Mary likes” forms 
a perfeelly aeeepfable fheme, mosf linguisls would 
agree fhaf if does nol also eomprise a valid grammal¬ 
ieal eonsliluenl. In spile of Ibis elaim, Ihough, if is 
interesting to note fhaf a number of ealegorial gram¬ 
mars, ineluding Combinalory Calegorial Grammar 
(CCG) (Sleedman, 2001), Ireal lexl segmenls like 
fhe above as possible synlaelie eonsliluenls. Con¬ 
sider fhe following dilransilive senfenee: 

^Actually many authors use the term focus as a synonym for 
rheme; the definitions we give in this paper follow (Steedman, 
2000 ). 



(5) John gave Mary a flower 

In CCG, this sentenee has a number of different 
syntaetie derivations, two of them are the following: 

John gave Mary a flower 

NP ((s\np)/np)/np np np 

(s\np)/np ^ (9) 

- > 

s\np 


John gave Mary a flower 

NP ((s\np)/np)/np np np 

S/(s\Nlh (s\np)/np ^ (10) 

- 

S/NP 


Note that (9) proeeeds by first eomposing the 
part eorresponding to the verb phrase (“gave Mary 
a flower”); later, in the final step, the verb phrase is 
eomposed with the subjeet ‘John’. The situation is 
reversed for (10), where the use of type-raising and 
eomposition rules of CCG allow the eonstruetion of 
the fragment “John gave Mary” as valid grammat- 
ieal text eonstituent, whieh is later eombined with 
the direet objeet of the sentenee (‘a flower’). Steed- 
man (2000) argues that this form of different syntae¬ 
tie derivations that one ean get even for very sim¬ 
ple sentenees when using CCG (some times referred 
to with the somewhat belittling term “spurious read¬ 
ings”), aetually serve to refleet variations in infor¬ 
mation strueture. Eaeh one of the above deriva¬ 
tions subsumes a different intonational pattern, dis¬ 
tinguishing the rheme from the theme when the sen¬ 
tenee is used for answering different questions: (9) 
answers to “Who gave Mary a flower?”, whereas 
(10) to “What did John give to Mary?”. 

In other words, the elaim here is that (a) surfaee 
strueture and information strueture eoineide; and (b) 
the role of information strueture is to provide a par- 
tieular interpretation of the surfaee strueture. Let us 
define this important idea in a preeise way, sinee it 
will be the eornerstone of the model presented in this 
paper: 

Postulate 4.1 Intonational boundaries in an utter¬ 
ance determine the intended syntactic structure. 

In our grammatieal formalism, pregroup gram¬ 
mars, variations in a grammatieal derivation similar 
to above are only implieitly assumed, sinee the or¬ 
der of eomposition remains unspeeified. This faet is 


apparent in the pregroup derivation of the example 
sentenee, where both (9) and (10) are subsumed into 
the following reduction diagram: 

John gave Mary a flower 
n rC s rM n n (11) 



Furthermore, it is direetly refleeted in our seman- 
tie spaee through the funetorial passage, via the faet 
that tensor eontraetion is assoeiative: 

John X (gave x Mary x flower) = (12) 

(John X gave x Mary) x flower 

Eq. 12 eonstitutes a natural manifestation of the 
principle of combinatory transparency (Steedman, 
2001): no matter in what order the various text eon- 
stituents are eombined, the semantie representation 
assigned to the sentenee is always the same; in other 
words, information strueture should not affeet se¬ 
mantie eonditions. Note, however, that even in the 
striet setting of formal semanties this is not always 
the ease. Consider the behaviour of the following 
sentenee under the presenee of the foeus-sensitive 
partiele ‘only’: 

(6) a. John only gave Mary A FLOWER 
b. John only gave Mary a flower 

The use of different intonational foeus elearly 
ehanges the semantie value of the sentenee: (6a) is 
true if the only thing that John gave to Mary was a 
flower (but he might have given things to other girls 
as well), while (6b) is true if the only person who 
got a flower from John was Mary. 

In the more relaxed and quantitative setting of a 
eompositional distributional model of meaning, the 
idea of having veetorial representations of words and 
sentenees that refleet intonational patterns seems 
even more legitimate. This eoneept is aligned with 
the distributional nature of sueh models: given a 
text eorpus eontaining information strueture anno¬ 
tations (of any kind), we would assume that the 
eo-oeeurrenee veetor of a word under foeus (say, 
BOO^) would slightly differ from that of the vee¬ 
tor representing the normal use of the word (book).^ 
Furthermore, we would expeet that, after the eom¬ 
position, this differenee would be also refleeted in 

^In the trivial case, this would be true by the presence of 
intonational markers in the immediate context of a word under 
focus, as opposed to its normal use. 



the vector representing the meaning of the entire 
sentence. From the next section we start working 
towards imposing this behaviour on the categorical 
model of Coecke et al. (2010). 

5 Intonation in pregroups 

Traditionally, a notational system describing intona¬ 
tion consists of markings that indicate pitch accents 
and boundaries. Using the notation of Pierrehum- 
bert and Hirschberg (1990), for example, we get the 
following for our example sentence:"*’^ 

(7) [MaryJr [likes musicalsJt 

H* L L+n* LH% 

The prosody starts with a sharp pitch accent (H*) 
that puts the focus on ‘Mary’, and continues with 
a rapid fall to low pitch (L boundary) that signifies 
a transition from rheme to theme. Within theme 
now, the focus goes to ‘musicals’ which gets the 
less rapidly rising pitch L-i-H*, whereas the bound¬ 
ary LH% expresses a rising continuation that marks 
the end of theme. In the case that theme precedes 
the rheme, we have the following pattern: 

(8) [Mary likeslx [musicalsJr 

L+n* LH% H* LL% 

As mentioned earlier, this paper mainly addresses 
the rheme/theme aspect of information structure, 
which is directly related to boundary markings. We 
start by representing an intonational boundary using 
a special token >, for which the following relations 
hold: 

theme > rheme or rheme <\ theme (13) 

Naturally, > is equivalent to LH%, while <i cor¬ 
responds to L in the Pierrehumbert and Hirschberg 
(1990) notation. It is very important to emphasize 
at this point that the above introduced tokens are far 
from an ad-hoc means for achieving a goal. Recall 
from our discussion in Section 3 that while in En¬ 
glish the means of imposing information structure is 
purely phonological, this is not necessarily the case 
for other languages. As a concrete case, in Buli (a 
Gur language spoken in Ghana), the rheme is pre¬ 
ceded by a focus marker, which again can be inter¬ 
preted as an information-structural boundary since it 

‘'Example taken from (Steedman, 2000). 

^From now on we explicitly mark themes and rhemes in 
our examples for clarity. 


separates the theme from rheme; this is shown in the 
following example (Fiedler et ah, 2006): 

(9) Q: What did the woman eat? 

A: 5 r)5b ka tue 
3sg eat (FM) beans 

To formalize this mixing of syntactical and infor¬ 
mation structure in the context of a pregroup gram¬ 
mar, we add the two boundary markers to the vocab¬ 
ulary and introduce two new atomic types: 

Theme: 6 Rheme: p (14) 

An intonation pregroup grammar then will have 
the following form: 

P(S U {>,<]}, {n,s,6»,/)}) 

For the case of a simple transitive sentence, we 
get the following boundary types, based on the fact 
that now the boundary (and not the verb) becomes 
the head of our sentence: 

-s-p^ <3 : / • s • 0' (15) 

The type dictionary changes accordingly: a tran¬ 
sitive verb such as ‘like’ will be assigned two more 
types rf ■ 9 and 0 ■ depending whether it pro¬ 
duces a left-hand theme or a right-hand theme in the 
sentence; similarly, nouns will be assigned the extra 
type p. For the two cases of Eq. 13, we obtain the 
following derivations: 

Mary likes > musicals 
n -rf e p (16) 



Mary < likes musicals 
p p^ s 6^ 9 n} n (17) 



After transferring this to FVect via our functor 
in Eq. 4, and extending its action on atomic types 
by defining J^{9) = 0 and J^{p) = P, we get the 
obvious semantic counterpart: 

Mary likes > musicals 

4 ^ ( 18 ) 

N e e’’5'P* p 



There are some important observations based on 
the derivation in (18) above. Firstly, our simple sen¬ 
tence now is given in terms of a theme and a rheme. 



as required, both of whieh eontribute equally to its 
eonstruetion. Additionally, note that our verb is not 
any more a funetion of two arguments (of a subjeet 
and an objeet) as in the eanonieal ease, but of a sin¬ 
gle noun: it takes as input a subjeet in order to return 
a theme. Henee, in eontrast to a typieal ease of a 
transitive verb, the semantie representation of whieh 
requires a tensor of order 3, in this ease the eorre- 
sponding linear map takes the form N ^ whieh 
ean be eanonieally represented by a matrix N 

The question of how to properly model intona¬ 
tion in eompositional distributional semanties is ev¬ 
idently epitomized in ehoosing an appropriate form 
for the tensor of the i> token in (18). In order to pro¬ 
vide an answer to this, we first need to examine the 
eoneepts of rheme and theme from a semantie point 
of view. 

6 A semantic truth-theoretic argument 

We use as an example the following simple ease: 

(10) Q: Who does John like? 

A: [John likesjj [MaryJr 

From an extensional point of view, the semantie 
value of the theme ean be seen as a set of alternative 
options (Rooth, 1992), eaeh one of whieh may be 
used as a response to the given question: 

[John (might) like] = {x|John (might) like x} 

As a eonsequenee, the role of the rheme now is 
to restrict the set of alternatives to a speeifie ehoiee 
(Steedman, 2000). Note that this aetion of restriet- 
ing the available ehoiees is responsibility of the into- 
national boundary; indeed, the boundary ean be seen 
as a binary operator that performs the merging of the 
theme with the rheme, restrieting the alternatives set 
of the theme to a speeifie response: 

(11) [John likesjj > [MaryJr := 

>([John likesjj,[M aryJr) 

This is what Diagram (18) shows; in our multi¬ 
linear setting, the boundary beeomes a bi-linear map 
0 (g) P ^ S' that performs the required “restrietion”. 
Now, what is the most appropriate way to model this 
operation in the extensional setting diseussed above? 
Note that by simply eheeking if rheme is eontained 
in the alternatives set is not suffieient; this would 
return true or false as an answer to a question that 
expeets a person. A more appropriate ehoiee then 


is to model the boundary by using set interseetion: 
we take the meaning of rheme to be a singleton that 
eontains the answer, and the meaning of the sentenee 
to be the interseetion of rheme with theme: 

{Mary} n {x|John (might) like x} (19) 

The answer will be again the singleton {Mary} if 
Mary is ineluded in the set of people who John po¬ 
tentially likes, and the empty set otherwise. Thus 
we have aehieved our goal: the theme set has 
been restrieted aeeording to the provided response. 
We generalize this argument to an arbitrary pair of 
rheme and theme (with 5theme denoting theme’s eor- 
responding alternative set) as follows: 

c>(rheme, theme) = {rheme} n Ptheme 

J rheme if rheme G Ptheme 
{ 0 o.w. 

6.1 From sets to vector spaces 

We transfer the above reasoning to veetor spaees, by 
eneoding sets and relations in veetorial forms. The 
veetorial form of a set is a veetor spaee (let it be 
N = {ni}i) whose basis veetors are the elements 
of the set. For the sake of demonstration (and this 
will beeome clear as the section reads on), we define 
our sentence space to be a one dimensional space 
where the origin denotes falsity and everything else 
denotes truth. One can take this to be a dimension 
in any vector space; here we take it to be in N and 
denote its basis vector with a basis vector of N. Fur¬ 
thermore, a binary relation such as likes{x,y) can 
be represented as an adjacency matrix W in which 
Wij is 1 if the pair {i,j) is contained in the relation 
and 0 otherwise. Note that this matrix is isomorphic 
to a tensor in N iSi S ® N, since our sentence space 
is one-dimensional. 

Let us apply categorical composition to compute 
a vectorial representation for the theme of our sen¬ 
tence, “John likes”. 


(CiV ® Is) 




ij ij 3 


( 21 ) 


Hence the vectorial representation of “John likes” 
becomes indeed the subset of all individuals who 
might be liked by the person denoted by vector nl, 
and can be seen as the semantic value of the theme 



of our sentence. The next step is to compose this 
theme with the rheme ‘Mary’; in other words, we 
must decide an appropriate type of composition for 
our intonational boundary. Let us first try again stan¬ 
dard categorical composition: 


(Is ® e(v) 



E ^3, K lni) = E WsjS,, = VL31 

j 3 


( 22 ) 


Note that this corresponds to a set membership 
test; the result is 1 if Mary is included in the set of 
alternative responses and 0 otherwise. However, as 
noted before, in information structure terms a more 
appropriate operation would be to take the intersec¬ 
tion of the singleton {Mary} with the set of alterna¬ 
tives. Interestingly, set intersection now corresponds 
to element-wise vector multiplication (in this work 
denoted by symbol ©) and the vector space equiva¬ 
lent of Eq. 19 becomes: 


= jr(0 . „') = ^ ^ 

Furthermore, boundaries are assigned to the fol¬ 
lowing vector space: 

-s- p^)= F{p^ ■ s-e^) = W (^W (^W 

We have now arrived at a central point of this pa¬ 
per. As the semantic representation of a boundary, 
we assign the following morphism: 


{Iw © Pw © Iw) ° ivw © Vw) (25) 

Note that the above is indeed an element in fL © 

W^W: 


0, -d : / = / © I 


3F^3}Z^W(E)W(E)W(E) 


W (26) 
W 


The reasoning behind our assignment will be¬ 
come clear in a moment. For now, we proceed to 
a formal definition: 


w f ^ iflTsi —1 

Ew^s.n, 1 ©ni = | ^ (23) 

The result is now ‘Mary’, if Mary is included in 
the set of valid answers, and the zero vector other¬ 
wise. The fact that the meaning of our sentence be¬ 
comes an element of the noun space demonstrates 
clearly that, in information structure terms, there is 
a necessity for a shared vector space between sen¬ 
tences and nouns (or noun phrases)—a direct con¬ 
sequence of the fact that now the meaning of a sen¬ 
tence is mainly focused on a specific noun or noun 
phrase fherein. Furfhermore, since a senfence is now 
expressed as a merging of a fheme and a rheme, if is 
also required fhaf Q = S = P (and equal fo whaf 
we look lo be N in fhe preceding). In fhe nexf sec- 
lion we encode fhe above reasoning in fhe abslracf 
form of compacl closed calegories and Ihen presenl 
an inslanlialion in vector spaces. 

7 Intonation in compact closed categories 
with Frobenius structure 

The poinf wilh regard to shared spaces is accom¬ 
plished by fhe following lypes assignmenl: 

T(x) = W Vx€ {n, s, e, p} (24) 

As a consequence of fhe above, fhe veclor spaces 
assigned fo Iransilive verbs are compuled as follows: 


Definition 7.1 The meaning vector of a sentence ex¬ 
pressed in information structure terms is given by: 

(Iw ® pw ® Ivr) ° ivw ® r]w){theme ® rheme) (27) 

when theme precedes the rheme, or as follows in the 
opposite case. 

(Iw (S’ pw (S Iw) ° (vw (S r]w)(rheme (S theme) (28) 
These vectors are depicted as follows: 



Note lhal fhe normal forms al fhe righl-hand side 
of fhe diagrams above are direcl applicalions of fhe 
Frobenius condition. Furfhermore, eilher fhe fheme 
or rheme here mighl correspond lo large lexl con- 
sliluenls, i.e. phrases or even sentences. In Ihis case, 
fhe proposed framework guarantees fhaf an appro¬ 
priate vector will be crealed for Ihem based on cale- 
gorical composilion. 



8 Vector space instantiation 

Our justification for using the semantic form of Eq. 
25 for the boundary comes from the fact that it pro¬ 
duces normal forms as below: 

^(theme (g) rheme) ^(rheme (g) theme) (31) 


This is exactly how element-wise vector multipli¬ 
cation is defined from a categorical perspective: 


/u(ui (g) V2) 


Ul U2 
© U2 = E V 


(32) 


As a result, the linear algebraic instantiations of 
Definition 7.1 become as follows: 


rheme 0 theme theme 0 rheme 


We stress again the fact that rheme and theme can 
have complex structures, and their vector meanings 
will reflect this strutter. For simple transitive sen¬ 
tences^ of the form “subject verb > object” or “sub¬ 
ject <1 verb object”, we get linear algebraic meanings 
as follows: 

(subj X verb) © obj subj © (verb X obj) 


As an example of a composed theme, consider: 


Mary likes ^ musicals Mary likes musicals 


W WW WWW w w ww w 




(33) 

A vector is computed for the theme ‘Mary likes’ 
according to the rules of the grammar, and then this 
vector is element-wise multiplied with the vector of 
the rheme (which, in this example, is just the distri¬ 
butional vector of the word). 


models—a term that emphasizes on the equal con¬ 
tribution of each word to the final result, which pro¬ 
duces a kind of average of the input vectors. Note 
that this behaviour stands in direct contrast with the 
categorical compositional approach, in which the 
type-logical identities of words strictly depend on 
their grammatical role. Due to their simplicity, vec¬ 
tor mixture models have been studied extensively 
(Mitchell and Lapata, 2008), demonstrating steady 
and reasonably good performance in a number of 
tasks. 

The significance of the Frobenius operators for 
our model (as opposed to some other form of com¬ 
binatory mechanism) is that their concrete manifes¬ 
tation in a vector space setting imposes exactly this 
vector mixture behaviour, in the form of element¬ 
wise vector multiplication. In other words, the result 
is a combination of two compositional approaches, 
vector mixtures and categorical models, in a unified 
framework: while categorical composition is still 
applied to compute vectorial representations for a 
theme and a rheme, the two parts contribute equally 
to the final result via element-wise multiplication 
imposed by the Frobenius operators. This puts the 
necessary focus on the appropriate part of the sen¬ 
tence, reflecting the variation in meaning intended 
by the intonational pattern. 

To what extent the notion of a rheme as a means 
for restricting the theme applies in FVect? Note 
that, from a geometric perspective, element-wise 
vector multiplication acts as a scaling of the basis; 
for example, (y) © ( 05 ) transforms the vector space 
in which the first vector lives so that the units on 
the x-axis are doubled and the units on the y-axis 
are halved.^ Furthermore, a zero value in one vector 
would completely eliminate the corresponding com¬ 
ponent in the other. Hence, the concept of restricting 
the theme has now taken a new quantitative form, 
generalizing appropriately our initial intuition (mo¬ 
tivated by set intersection) to the multi-dimensional, 
real-valued setting of FVect. 


9 Interpretation 

The transition from the set-theoretical framework to 
high dimensional real vector spaces poses the ques¬ 
tion what is the role of element-wise vector multi¬ 
plication in the latter setting. Compositional mod¬ 
els based on element-wise vector addition or mul¬ 
tiplication are usually referred to as vector mixture 

®We provide more complicated examples later in Sect. 11. 


10 Relation to previous work 

How does the above derivations correlate to the 
premises of the original framework, in which ‘likes’ 
is a transitive verb with type rf ■ Note that an¬ 
other application of the Frobenius condition on the 
normal form of Diagram (33) will give us: 

’of course we can think of a similar scaling taking place 
on the two axes of the second vector by factors x and y. 



likes 


Mary likes musicals Mary likes musicals 


4 ^ 4 ^ 4 " 4 " 4 ^ 4 " 

WWW = w ^ 


Y UrVJ 


A 

w w 
I A 


(34) 


In other words, the semantie representation of 
word ‘likes’ ean be still regarded as a bi-linear map, 
faithfully eneoded in a tensor of order 3, as required 
by the framework. In this ease, the tensor of ‘likes’ 
in FVect is seen as ereated by applying the mor¬ 
phism Iw ® on n matrix representing the verb 
‘likes’. The limitation, of eourse, is that now the 
middle wire earrying the result (the sentenee veetor 
spaee) eannot be any more differentiated from the 
two argument wires (the noun veetor spaees), sinee 
it is produeed by eopying one of them. 

Note that these are the Frobenius models of Kart- 
saklis et al. (2014), referred to as Copy-Subjeet and 
Copy-Objeet, and originally used as a means for 
faithfully eneoding a verb matrix to a tensor of order 
3, thus restoring the funetorial relation between the 
semantie representation and the grammatieal type. 
The present theory^ offers an alternative more eom- 
plete aeeount that goes far beyond providing a eon- 
venient way to expand a matrix to a eube. 


11 Covering complex intonational patterns 

So far we examined simple eases of intonation, in 
whieh our sentenee eonsisted of a single rheme and 
a theme. In this seetion we turn our attention to some 
more interesting examples. 

11.1 Multiple rhemes 

We will first examine the ease of a sentenee with 
more than one rhemes. Imagine the following ques¬ 
tion/answer dialogue: 

(12) Q: Who likes whom? 

A: [JohnJr [likeslx [MaryJr 

In our pregroup notation, this introduees two dis- 
tinet intonational boundaries in the sentenee. The 
derivation takes the following form: 

John <1 likes t> Mary 

p p^se‘ e e s p^ p ( 35 ) 

I I W 


*An early account of which also appears in the doctoral 
thesis of the first author (Kartsaklis, 2015). 


Note that the type of ‘likes’ now beeomes 9 ■ 6\ 
in other words, the theme is not any more a funetion 
(no adjoint is present in the type), but a higher or¬ 
der atomie entity. This is direetly refleeted in FVect 
where we get: 


John ^ likes ^ Mary 


r\A 

4 


4 


TJ 


tlTJ 


tJ 


John likes Mary 



(36) 


The result of this eomputation is now a matrix and 
not a veetor. Indeed, if we follow the linear algebraie 
ealculations we get: 


Yw ® //w)(John (g) likes (g) mar^) 


(John (g) Mary) 0 likes 


(37) 


The behaviour above follows the premises of the 
proposed model: Sinee our theme is a matrix, the 
caleulations follow naturally, produeing another ma¬ 
trix as the rheme (the tensor produet of the two indi¬ 
vidual rhemes) that restricts as required the theme 
via element-wise multiplieation. Note that this 
means that a sentenee with one rheme would not be 
eomparable with a sentenee with two rhemes, sinee 
it would live in a different spaee. That is again not 
surprising: the shape of theme defines the shape of 
the sentenee veetor spaee, and only themes of the 
same order ean be eompared to eaeh other. 


11.2 Relational words as rhemes 

We have eonveniently avoided to diseuss until now 
the case in which the rheme is not a noun phrase, but 
a relational word as below: 


(13) Q: How does John feel about Mary? 

A: [Johnjj [likesJr [Maryjj 

In pregroups we model such a situation by the fol¬ 
lowing derivation: 

John > likes <l Mary 

e e^sp^ pp p^so^ e (38) 

I I 

Note that this time the verb becomes a higher or¬ 
der rheme, getting the type p ■ p. However, when this 
is transferred to FVect the symmetry of the cate¬ 
gory and the commutativity of the Frobenius algebra 
means that the vector of the sentence becomes equal 
to that of Example (12). In general, problems due 
to commutativity of the Frobenius operators can be 
resolved if one moves to non-commutative versions 



of Frobenius algebras. Piedeleu et al. (2015) explore 
sueh eonstruetions in the eontext of language by ele¬ 
vating the eategorieal model of Coeeke et al. (2010) 
to an open quantum system setting, in whieh words 
are represented as mixed states. 

11.3 Nested rhemes 

Consider the following ease: 

(14) What was the book Mary wrote about? 
[Mary wrote a book about] j [artJr 


Mary wrote a book about art 



By applieation of the spider equality (Seetion 2.6) 
we get the normal form below, whieh eomputes a 
meaning for the sentenee as the element-wise multi- 
plieation of the veetors eomposed for the two themes 
with the veetor of the rheme: 


The interesting point here is that the intonational 
boundary is plaeed in a position that eonstitutes a 
glaring violation of the grammatieal strueture, whieh 
in the normal ease has the following form: 


Mary wrote a book about art 

n sn} n n^nn’’ n 




(39) 


For eases like these we should reeall that our 
framework is entirely built on the assumption of 
Postulate 4.1: in the eontext of information strue¬ 
ture, intonational boundaries determine the intended 
syntaetieal strueture. For our ease, we get: 


Mary wrote a book about > art 



The linear-algebraie result follows trivially as the 
usual element-wise eomposition of the theme with 
the rheme. 

11.4 Rheme in the middle of sentence 

In many eases a noun phrase ean serve as the rheme 
while being plaeed in the middle of the sentenee, 
splitting the theme into two parts: 

(15) Did Mary write an essay about art? 

[Mary wrotejj [A BOOkJr [about artjj 

In these eases, the left-hand intonational boundary 
gets the type • p • as below: 

Mary wrote > a book <l about art 
n 6 9^ p p p^ s 9^ 6rP' n (41) 

^ I w 

In other words, a new rheme is produeed that is 
used as input to the right-hand intonational bound¬ 
ary. In Fvect we get the following interaetion: 


Mary wrote a book about art 



12 Conclusion and future work 

The present paper provides a first aeeount of into¬ 
nation and information strueture for the emerging 
field of eafegorieal eomposifional disfribufional se- 
manfies. In a more generie level, if lays fhe ground¬ 
work for a model eapable of aeeommodafing fwo 
differenl fypes of eomposition over a disfribufional 
selling. An experimenfal evalualion is deferred for 
fhe fufure, preferably in fhe eonlexl of a queslion- 
answering lask. There is also a lol of inferesling 
work fo be done on fhe Iheory side. Al fhe eur- 
renl slage, for example, fhe semanlic value of in- 
fonalional boundaries is given by direel assignmenl 
of a speeifie morphism—a eommon praeliee in fhe 
pasl for fhe relevanl liferalure. A fulure direelion, 
Ihen, more aligned wilh fhe eafegorieal nalure of fhe 
model, would be lo embed fhe appropriale Iransla- 
fion info fhe funelorial passage ilself. This ehalleng- 
ing goal requires novel Iheorelieal eonlribulions lhal 
will elevate fhe eoneepf of a pregroup grammar fo a 
new enlify equipped wilh Frobenius slruelure. 

Finally, fhe eafegorieal eomposifional model of 
Piedeleu ef al. (2015) is very relevanl fo our inler- 
esls, sinee if ean aeeommodafe a variety of non- 
eommulalive Frobenius algebras fhe linguisfie infu- 
ifion of whieh in relalion lo Ihis work remains lo be 
explored. 
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